Free your Camera: 3D Indoor Scene Understanding from Arbitrary Camera Motion

Many works have been presented for indoor scene understanding, yet few of them combine structural reasoning with full motion estimation in a real-time oriented approach. In this work we address the problem of estimating the 3D structural layout of complex and cluttered indoor scenes from monocular video sequences, where the observer can freely move in the surrounding space. We propose an effective probabilistic formulation that allows to generate, evaluate and optimize layout hypotheses by integrating new image evidence as the observer moves. Compared to state-of-the-art work, our approach makes significantly less limiting hypotheses about the scene and the observer (e.g., Manhattan world assumption, known camera motion). We introduce a new challenging dataset and present an extensive experimental evaluation, which demonstrates that our formulation reaches near-real-time computation time and outperforms state-of-the-art methods while operating in significantly less constrained conditions. Figure 1 shows a pictorial representation and a schematic diagram of the whole process. Sparse 3D reconstruction. As the observer moves in the surrounding environment, we first pre-process sequences with a localization and sparse 3D reconstruction algorithm. In our experiments we compare two such approaches: a real-time implementation of the Monocular V-SLAM approach proposed in [4] and the non-real-time VisualSfM [6]. These 3D reconstructions are in general noisy and sparse. Candidate layout components. The second step consists of generating a higher level representation of the 3D points estimated in the preprocessing phase. Several types of geometrical primitives are suitable for this purpose. In our case, we believe a piecewise planar representation is the most appropriate for indoor scene representation. We fit a large number of planes to the 3D points so as to generate a large number of (potentially inaccurate) candidates of layout components, i.e. walls, floor, ceiling. In our experiments we implemented an Iterative RanSaC plane fitting procedure, which we optimized for indoor scenes by allowing peripheral fitted points to be re-injected in the iteration process, since these points potentially lay on the intersection of two planes. Layout Parametrization While much prior work has leveraged the Manhattan world assumption, we believe this is a limiting hypothesis. To overcome this limitation, in this paper we adopt a representation similar to [5] (sometimes referred to as Soft Manhattan), which makes the following assumptions about the environment: i) ground plane and ceiling are parallel; ii) walls are only constrained to be orthogonal to the ground plane (and ceiling); iii) there can be any number of walls and each wall can be displaced at any angle with respect to other walls. Layout estimation. In the last step, constituting the core of our proposed inference engine, we generate layout hypotheses as random combinations of candidate layout components. Each layout hypothesis is evaluated at each time frame by measuring its compatibility with observations (e.g. image points and lines) and geometrical constraints across frames. During this process, each layout is “perturbed” by locally adjusting, optimizing, merging or splitting layout components. There are different approaches to manage sets of hypotheses. In this paper we choose to integrate our probabilistic framework within a particle filter structure. This choice allows to explicitly formulate the problem in a parallel-computing oriented fashion (particles are independent from each other), which can lead to high efficiency gains in computation time. The output of the optimization procedure is an estimation of the 3D scene layout, which is obtained by selecting the layout hypothesis with the best set of layout components. Figure 1: 3D scene layout estimation process. The video sequence is first processed to obtain camera localization and sparse 3D point cloud reconstruction. Layout components (e.g. floor, ceiling, walls) are generated from the sparse 3D points and combined to generate layout hypotheses. Each layout hypothesis is evaluated and optimized by incorporating new image evidence. The final 3D scene layout is represented by the hypothesis that better describes the scene.